{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 通过 TimedRotatingFileHandler 按时间切割日志\n",
    "\n",
    "> 线上跑了一个定时脚本，每天生成的日志文件都写在了一个文件中。但是日志信息不可能输出到单一的一个文件中。  \n",
    "> 原因有二：1. 日志文件越来越大会影响系统的性能。2. 日志文件格式不够清晰，比如我想看今天的日志，不太方便找到的今天的日志信息 (即使对日志输出做了时间提示)  \n",
    "> 通过设置`TimedRotatingFileHandler`进行日志按**周 (W)、天(D)、时(H)、分(M)、秒(S)** 切割。  \n",
    "\n",
    "先看一个简单例子："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```py\n",
    "import time\n",
    "import logging\n",
    "import os\n",
    "from logging import handlers\n",
    "\n",
    "\n",
    "def _logging(**kwargs):\n",
    "    level = kwargs.pop('level', None)\n",
    "    filename = kwargs.pop('filename', None)\n",
    "    datefmt = kwargs.pop('datefmt', None)\n",
    "    format = kwargs.pop('format', None)\n",
    "    if level is None:\n",
    "        level = logging.DEBUG\n",
    "    if filename is None:\n",
    "        filename = 'default.log'\n",
    "    if datefmt is None:\n",
    "        datefmt = '%Y-%m-%d %H:%M:%S'\n",
    "    if format is None:\n",
    "        format = '%(asctime)s [%(module)s] %(levelname)s [%(lineno)d] %(message)s'\n",
    "\n",
    "    log = logging.getLogger(filename)\n",
    "    format_str = logging.Formatter(format, datefmt)\n",
    "    # backupCount 保存日志的数量，过期自动删除\n",
    "    # when 按什么日期格式切分(这里方便测试使用的秒)\n",
    "    th = handlers.TimedRotatingFileHandler(filename=filename, when='S', backupCount=3, encoding='utf-8')\n",
    "    th.setFormatter(format_str)\n",
    "    th.setLevel(logging.INFO)\n",
    "    log.addHandler(th)\n",
    "    log.setLevel(level)\n",
    "    \n",
    "    # 为了看的更视觉效果，可以显示在控制台答应\n",
    "    cmd = logging.StreamHandler()\n",
    "    cmd.setFormatter(format_str)\n",
    "    cmd.setLevel(level)\n",
    "    log.addHandler(cmd)\n",
    "    \n",
    "    return log\n",
    "\n",
    "\n",
    "os.makedirs(\"./logs\", exist_ok=True)\n",
    "logger = _logging(filename='./logs/default.log')\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    while True:\n",
    "        time.sleep(0.1)\n",
    "        logger.info('哈哈哈')\n",
    "```\n",
    "\n",
    "结果如下：  \n",
    "![](../assets/logging1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 参数\n",
    "\n",
    "- filename：日志文件名的prefix；\n",
    "- when：是一个字符串，用于描述滚动周期的基本单位，字符串的值及意义如下：\n",
    "“S”: Seconds  \n",
    "“M”: Minutes  \n",
    "“H”: Hours  \n",
    "“D”: Days  \n",
    "“W”: Week day (0=Monday)  \n",
    "“midnight”: Roll over at midnight  \n",
    "- interval: 滚动周期，单位有when指定，比如：when=’D’,interval=1，表示每天产生一个日志文件；\n",
    "- backupCount: 表示日志文件的保留个数；\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义日志文件名\n",
    "\n",
    "> 上述代码可以正常运行，而且也可以生成固定的日志个数，但是有一个问题，生成的日志文件格式是你的`文件名+时间`的格式，没有设置时间的话默认设置到了秒 (这里是按秒切割)  \n",
    "\n",
    "修改日志文件名格式\n",
    "\n",
    "### 自定义日期后缀格式\n",
    "\n",
    "除了上述参数之外，`TimedRotatingFileHandler`还有两个比较重要的成员变量，它们分别是`suffix`和`extMatch`。`suffix`是指日志文件名的后缀,`suffix`中通常带有格式化的时间字符串，filename和suffix由“.”连接构成文件名（例如：`filename=“runtime”， suffix=“%Y-%m-%d.log”`,生成的文件名为runtime.2015-07-06.log）。\n",
    "\n",
    "```py\n",
    "# 设置为S，默认的suffix为 Y-%m-%d_%H-%M-%S\n",
    "filehandler.suffix = \"%Y-%m-%d_%H-%M-%S.log\"\n",
    "```\n",
    "\n",
    "### 自定义整个日志文件名格式（❗有坑、慎用）\n",
    "\n",
    "```py\n",
    "# 在上述代码中加入一个函数：\n",
    "def namer(filename):\n",
    "    # filename参数为日志文件的绝对路径\n",
    "    # 返回重命名后的文件绝对路径\n",
    "    return filename.replace('default.log.', '')\n",
    "```\n",
    "\n",
    "然后给handler的namer 属性赋值:\n",
    "\n",
    "```py\n",
    "filehandler.namer = namer\n",
    "```\n",
    "\n",
    "运行结果：  \n",
    "![](../assets/logging2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 名字好像可以了，**但是日志好像没有起到自动删除的目的**, 来看看源码：\n",
    "\n",
    "```py\n",
    "def getFilesToDelete(self):\n",
    "        \"\"\"\n",
    "        Determine the files to delete when rolling over.\n",
    "\n",
    "        More specific than the earlier method, which just used glob.glob().\n",
    "        \"\"\"\n",
    "        dirName, baseName = os.path.split(self.baseFilename)\n",
    "        fileNames = os.listdir(dirName)\n",
    "        result = []\n",
    "        prefix = baseName + \".\"\n",
    "        plen = len(prefix)\n",
    "        for fileName in fileNames:\n",
    "            if fileName[:plen] == prefix:\n",
    "                suffix = fileName[plen:]\n",
    "                if self.extMatch.match(suffix):\n",
    "                    result.append(os.path.join(dirName, fileName))\n",
    "        if len(result) < self.backupCount:\n",
    "            result = []\n",
    "        else:\n",
    "            result.sort()\n",
    "            result = result[:len(result) - self.backupCount]\n",
    "        return result\n",
    "```\n",
    "\n",
    "> 这是它的删除逻辑，是通过查找`self.baseFilename`同一目录下的，带有filename（我此处为default.log）前缀的文件列表，按时间从大到小排序后取，返回大于 `backupCount` 数量的要删除的文件名列表。\n",
    "\n",
    "由于后来修改文件名格式时把文件名前缀`default.log.`给替换为空，没有了了公共前缀，便不能删除过量的日志文件了。所以在 namer 属性定义的函数中，一定要保留baseFilename（default.log)字符串。\n",
    "\n",
    "### 删除日志设置：extMatch属性\n",
    "\n",
    "extMatch是一个编译好的正则表达式，用于匹配日志文件名的后缀，一般来说它和suffix是匹配的，如果extMatch匹配不到日志文件名的话，过期的日志是不会被删除的。比如，suffix=“%Y-%m-%d.log”, extMatch的只应该是re.compile(r”^\\d{4}-\\d{2}-\\d{2}.log$”)。默认情况下，在TimedRotatingFileHandler对象初始化时，`suffix`和`extMatch`会根据when的值进行初始化。\n",
    "\n",
    "| when     | suffix             | extMatch                                |\n",
    "| :------- | :----------------- | :--------------------------------------- |\n",
    "| S’       | %Y-%m-%d\\_%H-%M-%S | r\"\\^d{4}-\\d{2}-\\d{2}\\_\\d{2}-\\d{2}-\\d{2}\" |\n",
    "| M        | %Y-%m-%d\\_%H-%M    | r\"^\\d{4}-\\d{2}-\\d{2}\\_\\d{2}-\\d{2}\"       |\n",
    "| H        | %Y-%m-%d\\_%H       | r\"^\\d{4}-\\d{2}-\\d{2}\\_\\d{2}\"             |\n",
    "| D        | %Y-%m-%d           | r\"^\\d{4}-\\d{2}-\\d{2}\"                    |\n",
    "| MIDNIGHT | %Y-%m-%d           | r\"^\\d{4}-\\d{2}-\\d{2}\"                    |\n",
    "| W        | Y-%m-%d”           | r\"^\\d{4}-\\d{2}-\\d{2}\"                    |\n",
    "\n",
    "**如果对日志文件名没有特殊要求的话，可以不用设置suffix和extMatch**，如果需要修改了suffix，也一定要修改extMatch，让它们匹配上。\n",
    "\n",
    "例如：\n",
    "\n",
    "```py\n",
    "filehandler.suffix = \"%Y-%m-%d_%H-%M.log\"\n",
    "filehandler.extMatch = re.compile(r\"^\\d{4}-\\d{2}-\\d{2}_\\d{2}-\\d{2}.log$\")\n",
    "```\n",
    "\n",
    "该属性作用见上面的`getFilesToDelete`中\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```py\n",
    "import time\n",
    "import logging\n",
    "import os\n",
    "from logging import handlers\n",
    "\n",
    "\n",
    "def _logging(**kwargs):\n",
    "    level = kwargs.pop('level', None)\n",
    "    filename = kwargs.pop('filename', None)\n",
    "    datefmt = kwargs.pop('datefmt', None)\n",
    "    format_ = kwargs.pop('format', None)\n",
    "    if level is None:\n",
    "        level = logging.DEBUG\n",
    "    # filename 要保存的文件名\n",
    "    if filename is None:\n",
    "        filename = 'default.log'\n",
    "    # datefmt 时间格式\n",
    "    if datefmt is None:\n",
    "        datefmt = '%Y-%m-%d %H:%M:%S'\n",
    "    if format_ is None:\n",
    "        format_ = '%(asctime)s [%(name)s:%(lineno)d] [%(levelname)s]- %(message)s'\n",
    "\n",
    "    logger = logging.getLogger(__name__)\n",
    "    logger.setLevel(level)\n",
    "    formatter = logging.Formatter(format_, datefmt)\n",
    "\n",
    "    # 1.设置log日志文件按时间拆分记录，并保存几个历史文件，如果不需要拆分文件记录可忽略\n",
    "    # 例：设置每天保存一个log文件，以日期为后缀，保留3个旧文件。\n",
    "    filehandler = handlers.TimedRotatingFileHandler(filename=filename, when='S', backupCount=3, encoding='utf-8')\n",
    "    filehandler.suffix = \"%Y-%m-%d_%H-%M-%S.log\"  # 设置历史文件 后缀\n",
    "    filehandler.setFormatter(formatter)\n",
    "    filehandler.setLevel(logging.INFO)\n",
    "    logger.addHandler(filehandler)\n",
    "\n",
    "    # 2.设置log日志的标准输出打印，如果不需要在终端输出结果可忽略\n",
    "    console = logging.StreamHandler()\n",
    "    console.setFormatter(formatter)\n",
    "    console.setLevel(level)\n",
    "    # 只为当前logger启用console\n",
    "    logger.addHandler(console)\n",
    "\n",
    "    # 所有日志都启用console\n",
    "    # logging.getLogger('').addHandler(console)\n",
    "\n",
    "    return logger\n",
    "\n",
    "\n",
    "os.makedirs(\"./logs\", exist_ok=True)\n",
    "logger = _logging(filename='./logs/default.log')\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    while True:\n",
    "        time.sleep(0.1)\n",
    "        logger.info('哈哈哈')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "237.818px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
